Hierarchical multimodal transformer to summarize videos

Although video summarization has achieved tremendous success benefiting from Recurrent Neural Networks (RNN), RNN-based methods neglect the global dependencies and multi-hop relationships among frames, which limits performance. Transformer is an effective model to deal with this problem, surpasses in several sequence modeling tasks, such as machine translation, captioning, etc. Motivated by great of transformer natural structure (frame-shot-video), a hierarchical developed for summarization, can capture frame shots, summarize exploiting scene information formed shots. Furthermore, we argue that both audio visual are essential task. To integrate two kinds information, they encoded two-stream scheme, multimodal fusion mechanism based on transformer. In paper, proposed method denoted Hierarchical Multimodal (HMT). Practically, extensive experiments show HMT achieves (F-measure: 0.441, Kendall’s ?: 0.079, Spearman’s ?: 0.080) 0.601, 0.096, 0.107) SumMe TVsum, respectively. It most traditional, attention-based methods.